一个用于Plone导出和导入内容、成员、关系、翻译和本地角色的附加组件。
项目描述
collective.exportimport
导出和导入内容、成员、关系、翻译、本地角色等。
使用中间json格式从和到Plone站点导出和导入各种数据。主要用例是迁移,因为它允许您在一步之内从使用Archetypes和Python 2的Plone 4迁移到使用Dexterity和Python 3的Plone 6。大多数功能使用plone.restapi来序列化和反序列化数据。
有关使用exportimport迁移的培训,请参阅:https://training.plone.org/migrations/exportimport.html
功能
导出 & 导入内容
导出 & 导入成员和组及其角色
导出 & 导入关系
导出 & 导入翻译
导出 & 导入本地角色
导出 & 导入顺序(在父项中的位置)
导出 & 导入讨论/评论
导出 & 导入版本化内容
导出 & 导入重定向
导出支持
Plone 4, 5 和 6
Archetypes 和 Dexterity
Python 2 和 3
plone.app.multilingual, Products.LinguaPlone, raptus.multilanguagefields
导入支持
Plone 5.2+, Dexterity, Python 2 和 3, plone.app.multilingual
安装
像安装任何其他 Python 包一样安装 collective.exportimport。
您不需要在站点设置附加组件控制面板中激活附加组件,就能在您的网站上使用 @@export_content 和 @@import_content 表单。
如果您需要帮助,请参阅:- 对于 Plone 4: https://4.docs.plone.org/adapt-and-extend/install_add_ons.html - 对于 Plone 5: https://5.docs.plone.org/manage/installing/installing_addons.html - 对于 Plone 6: https://6.docs.plone.org/install/manage-add-ons-packages.html
Python 2兼容性
此包与 Python 3 和 Python 2 兼容。根据 Python 版本,将安装其依赖项的不同版本。如果您遇到问题,请在:https://github.com/collective/collective.exportimport/issues 上提交问题
用法
导出
使用 URL /@@export_content 的表单,并选择您想要导出的内容
您可以导出一种或多种类型,整个站点或仅站点中的特定路径。由于项目是按路径顺序导出的,因此导入它们将创建与最初相同的结构。
下载的 json 文件将具有您导出的路径的名称,例如 Plone.json。
成员、关系、本地角色和关系的导出链接已在此表单中列出,但也可以单独调用:/@@export_members、/@@export_relations、/@@export_localroles、/@@export_translations、/@@export_ordering、/@@export_discussion。
导入
使用URL为/@@import_content的表单,上传您想要导入的json文件
成员、关系、本地角色和关系的导入链接已在此表单中列出,但也可以单独调用:/@@import_members、/@@import_relations、/@@import_localroles、/@@import_translations、/@@import_ordering、/@@import_discussion。
在迁移的最后一步,还有一个名为@@reset_dates的视图,该视图将导入内容的修改日期重置为导入的json文件中最初包含的日期。这是必要的,因为在迁移过程中,各种更改可能会导致修改日期的更新。在导入过程中,原始日期作为每个新对象的obj.modification_date_migrated存储,此视图设置此日期。
导出和导入位置
如果您选择“保存到服务器上的文件”,导出视图将保存json文件到您的Plone实例的<var>目录中,位于/var/instance。导入视图将在/var/instance/import下查找文件。这些目录通常在不同Plone实例中,可能在不同的服务器上。
您可以将环境变量‘COLLECTIVE_EXPORTIMPORT_CENTRAL_DIRECTORY’设置为在服务器上或可能网络上添加一个“共享”目录。设置此变量后,collective.exportimport将同时保存到和从同一服务器目录加载.json文件。这样可以节省时间,不需要在导出和导入位置之间移动.json文件。请注意,导出视图将覆盖任何具有相同名称的现有以前的.json文件导出。
用例
迁移
如果不需要就地迁移,可以选择此插件将网站最重要的部分迁移到json,然后将其导入到您目标版本的新Plone实例中
从Plone站点导出内容(它支持Plone 4和5,Archetypes和Dexterity,Python 2和3)。
将导出的内容导入到新站点(Plone 5.2+,Dexterity,Python 3)
导出和导入关系、用户和组及其角色、翻译、本地角色、排序、默认页面、评论、小部件和重定向。
有关如何迁移附加功能(如注释或标记接口)的讨论,请参阅FAQ部分。
其他
您可以使用此插件来
将内容存档为JSON。
导出数据以准备迁移到另一个系统。
将多个plone站点的内容合并到一个。
将plone站点作为子站点导入到另一个。
导入其他系统的内容,只要它符合所需格式。
更新或替换现有数据。
详细信息
导出内容
导出内容基本上是plone.restapi序列化器的包装器
from plone.restapi.interfaces import ISerializeToJson
from zope.component import getMultiAdapter
serializer = getMultiAdapter((obj, request), ISerializeToJson)
data = serializer(include_items=False)
导入内容
导入内容是plone.restapi反序列化器的复杂包装器
from plone.restapi.interfaces import IDeserializeFromJson
from zope.component import getMultiAdapter
container.invokeFactory(item['@type'], item['id'])
deserializer = getMultiAdapter((new, self.request), IDeserializeFromJson)
new = deserializer(validate_all=False, data=item)
用于迁移
此包的主要用例是从一个Plone版本迁移到另一个版本。
将Archetypes内容导出并作为Dexterity内容导入可以正常工作,但由于字段名的更改,一些设置会丢失。例如,从导航中排除内容的设置已从excludeFromNav重命名为exclude_from_nav。
要修复此问题,您可以检查“修改导出数据以进行迁移”复选框。这将修改导出过程中的数据
删除未使用的数据(例如,next_item和components)
删除所有关系字段
更改Archetypes和Dexterity之间更改的一些字段名
excludeFromNav → exclude_from_nav
allowDiscussion → allow_discussion
subject → subjects
expirationDate → expires
effectiveDate → effective
creation_date → created
modification_date → modified
startDate → start
endDate → end
openEnd → open_end
wholeDay → whole_day
contactEmail → contact_email
contactName → contact_name
contactPhone → contact_phone
更新自Plone 4以来更改的文件夹和集合的视图名称。
将ATTopic及其标准导出到带有查询字符串的集合中。
更新集合标准。
内容和小部件的Richtext字段中的链接和图片自Plone 4以来已更改。视图/@@fix_html允许您修复这些问题。
控制导入内容的创建
您可以选择四种方式处理已存在的内容:
跳过:完全不导入
替换:删除项目并创建新的
更新:重用并仅覆盖导入的数据
忽略:使用新ID创建
导入的内容最初使用invokeFactory创建,使用导出项的portal_type和id,在反序列化其余数据之前。您可以通过指定一个字典factory_kwargs来设置附加值,该字典将被传递给工厂。这样,您可以设置导入对象上的值,这些值是IObjectAddedEvent订阅者预期的。
导出版本化内容
由于plone.restapi中的错误(https://github.com/plone/plone.restapi/issues/1335),导出Archetypes内容的版本将不起作用。要使导出工作,您需要使用7.7.0和8.0.0之间的版本(如果已发布)或7.x.x分支的源签出。
关于速度和大型迁移的说明
导出和导入大量内容可能需要一些时间。导出非常快,但导入受Plone某些功能的限制,最重要的是版本控制。
导入5000个文件夹需要约5分钟。
由于版本控制,导入5000个文档需要>25分钟。
在不进行版本控制的情况下导入5000个文档需要约7分钟。
在导入过程中,您可以每导入x个条目提交一次,这将释放TMPDIR(在提交之前在此处添加blob)中的内存和磁盘空间。
在导出大量blob(二进制文件和图像)时,您将得到巨大的json文件,可能会耗尽内存。您有多种处理此问题的方法。最佳方法取决于您如何导入blob。
作为下载URL导出:下载量小,但collective.exportimport无法导入blob,因此您需要自己的导入脚本来下载它们。
作为base-64编码的字符串导出:下载量大,但collective.exportimport可以处理导入。
导出为blob路径:小文件下载,collective.exportimport可以处理导入,但您需要将var/blobstorage复制到您进行导入的Plone站点,或者设置环境变量COLLECTIVE_EXPORTIMPORT_BLOB_HOME为旧的blobstorage路径:export COLLECTIVE_EXPORTIMPORT_BLOB_HOME=/path-to-old-instance/var/blobstorage。要导出blob路径,您不需要访问blob!
内容导出和导入的格式
默认情况下,所有内容都导出到一个大型json文件中,并从该文件导入。为了检查这样的大型json文件而不出现性能问题,您可以使用klogg (https://klogg.filimonov.dev)。
从版本1.10开始,collective.exportimport也支持将每个内容项作为单独的json文件导出和导入。要使用它,请在表单中选择将每个项目保存为服务器上的单独文件,或在调用导出时指定download_to_server=2。在导入表单中,您可以手动选择服务器上的目录或指定server_directory="/mydir",在调用导入时指定。
自定义导出和导入
此插件旨在根据您的需求进行调整,并具有多个挂钩,以使其易于实现。
为了使这更容易,以下是一些您可以用作覆盖和扩展导出和导入的包。使用这些模板并根据您自己的项目进行修改
以下章节收集了关于自定义导出和导入的许多示例。
导出示例
from collective.exportimport.export_content import ExportContent
class CustomExportContent(ExportContent):
QUERY = {
'Document': {'review_state': ['published', 'pending']},
}
DROP_PATHS = [
'/Plone/userportal',
'/Plone/en/obsolete_content',
]
DROP_UIDS = [
'71e3e0a6f06942fea36536fbed0f6c42',
]
def update(self):
"""Use this to override stuff before the export starts
(e.g. force a specific language in the request)."""
def start(self):
"""Hook to do something before export."""
def finish(self):
"""Hook to do something after export."""
def global_obj_hook(self, obj):
"""Inspect the content item before serialisation data.
Bad: Changing the content-item is a horrible idea.
Good: Return None if you want to skip this particular object.
"""
return obj
def global_dict_hook(self, item, obj):
"""Use this to modify or skip the serialized data.
Return None if you want to skip this particular object.
"""
return item
def dict_hook_document(self, item, obj):
"""Use this to modify or skip the serialized data by type.
Return the modified dict (item) or None if you want to skip this particular object.
"""
return item
将其与自己的browserlayer注册以覆盖默认值。
<browser:page
name="export_content"
for="zope.interface.Interface"
class=".custom_export.CustomExportContent"
layer="My.Custom.IBrowserlayer"
permission="cmf.ManagePortal"
/>
导入示例
from collective.exportimport.import_content import ImportContent
class CustomImportContent(ImportContent):
CONTAINER = {'Event': '/imported-events'}
# These fields will be ignored
DROP_FIELDS = ['relatedItems']
# Items with these uid will be ignored
DROP_UIDS = ['04d1477583c74552a7fcd81a9085c620']
# These paths will be ignored
DROP_PATHS = ['/Plone/doormat/', '/Plone/import_files/']
# Default values for some fields
DEFAULTS = {'which_price': 'normal'}
def start(self):
"""Hook to do something before importing one file."""
def finish(self):
"""Hook to do something after importing one file."""
def global_dict_hook(self, item):
if isinstance(item.get('description', None), dict):
item['description'] = item['description']['data']
if isinstance(item.get('rights', None), dict):
item['rights'] = item['rights']['data']
return item
def dict_hook_customtype(self, item):
# change the type
item['@type'] = 'anothertype'
# drop a field
item.pop('experiences', None)
return item
def handle_file_container(self, item):
"""Use this to specify the container in which to create the item in.
Return the container for this particular object.
"""
return self.portal['imported_files']
注册它
<browser:page
name="import_content"
for="zope.interface.Interface"
class=".custom_import.CustomImportContent"
layer="My.Custom.IBrowserlayer"
permission="cmf.ManagePortal"
/>
自动化导出和导入
运行所有导出并将所有数据保存到var/instance/
from plone import api
from Products.Five import BrowserView
class ExportAll(BrowserView):
def __call__(self):
export_content = api.content.get_view("export_content", self.context, self.request)
self.request.form["form.submitted"] = True
export_content(
portal_type=["Folder", "Document", "News Item", "File", "Image"], # only export these
include_blobs=2, # Export files and images as blob paths
download_to_server=True)
other_exports = [
"export_relations",
"export_members",
"export_translations",
"export_localroles",
"export_ordering",
"export_defaultpages",
"export_discussion",
"export_portlets",
"export_redirects",
]
for name in other_exports:
view = api.content.get_view(name, portal, request)
# This saves each export in var/instance/export_xxx.json
view(download_to_server=True)
# Important! Redirect to prevent infinite export loop :)
return self.request.response.redirect(self.context.absolute_url())
使用上面示例中的导出数据进行所有导入
from collective.exportimport.fix_html import fix_html_in_content_fields
from collective.exportimport.fix_html import fix_html_in_portlets
from pathlib import Path
from plone import api
from Products.Five import BrowserView
class ImportAll(BrowserView):
def __call__(self):
portal = api.portal.get()
# Import content
view = api.content.get_view("import_content", portal, request)
request.form["form.submitted"] = True
request.form["commit"] = 500
view(server_file="Plone.json", return_json=True)
transaction.commit()
# Run all other imports
other_imports = [
"relations",
"members",
"translations",
"localroles",
"ordering",
"defaultpages",
"discussion",
"portlets",
"redirects",
]
cfg = getConfiguration()
directory = Path(cfg.clienthome) / "import"
for name in other_imports:
view = api.content.get_view(f"import_{name}", portal, request)
path = Path(directory) / f"export_{name}.json"
results = view(jsonfile=path.read_text(), return_json=True)
logger.info(results)
transaction.commit()
# Run cleanup steps
results = fix_html_in_content_fields()
logger.info("Fixed html for %s content items", results)
transaction.commit()
results = fix_html_in_portlets()
logger.info("Fixed html for %s portlets", results)
transaction.commit()
reset_dates = api.content.get_view("reset_dates", portal, request)
reset_dates()
transaction.commit()
常见问题,技巧和窍门
本节涵盖了常见的使用场景和示例,这些功能不是所有迁移都需要的。
在导出期间使用 global_obj_hook
在导出期间使用global_obj_hook来检查内容并决定跳过它。
def global_obj_hook(self, obj):
# Drop subtopics
if obj.portal_type == "Topic" and obj.__parent__.portal_type == "Topic":
return
# Drop files and images from PFG formfolders
if obj.__parent__.portal_type == "FormFolder":
return
return obj
在导出期间使用 dict-hooks
在导出期间使用global_dict_hook来检查内容并修改序列化的json。您还可以使用dict_hook_<somecontenttype>来更好地结构您的代码以提高可读性。
有时您需要在导出期间的global_dict_hook中处理数据,并在导入期间对应的global_object_hook中的相应代码中处理这些数据。
以下关于地方性工作流策略的示例是这种模式的完美示例。
导出/导入 placeful 工作流程策略
导出
def global_dict_hook(self, item, obj):
if obj.isPrincipiaFolderish and ".wf_policy_config" in obj.keys():
wf_policy = obj[".wf_policy_config"]
item["exportimport.workflow_policy"] = {
"workflow_policy_below": wf_policy.workflow_policy_below,
"workflow_policy_in": wf_policy.workflow_policy_in,
}
return item
导入
def global_obj_hook(self, obj, item):
wf_policy = item.get("exportimport.workflow_policy")
if wf_policy:
obj.manage_addProduct["CMFPlacefulWorkflow"].manage_addWorkflowPolicyConfig()
wf_policy_config = obj[".wf_policy_config"]
wf_policy_config.setPolicyIn(wf_policy["workflow_policy_in"], update_security=True)
wf_policy_config.setPolicyBelow(wf_policy["workflow_policy_below"], update_security=True)
在导入期间使用 dict-hooks
您可以使用global_dict_hook或dict_hook_<contenttype>在导入期间执行许多修复。
我们在此处防止到期日期在有效日期之前,因为这会导致反序列化时的验证错误。
def global_dict_hook(self, item):
effective = item.get('effective', None)
expires = item.get('expires', None)
if effective and expires and expires <= effective:
item.pop('expires')
return item
我们在此处删除创建者中的空行。
def global_dict_hook(self, item):
item["creators"] = [i for i in item.get("creators", []) if i]
return item
此示例在导入期间将PloneHelpCenter迁移到简单的文件夹/文档结构。还有其他一些类型要处理(作为文件夹或文档),但您应该明白这个概念,对吗?
def dict_hook_helpcenter(self, item):
item["@type"] = "Folder"
item["layout"] = "listing_view"
return item
def dict_hook_helpcenterglossary(self, item):
item["@type"] = "Folder"
item["layout"] = "listing_view"
return item
def dict_hook_helpcenterinstructionalvideo(self, item):
item["@type"] = "File"
if item.get("video_file"):
item["file"] = item["video_file"]
return item
def dict_hook_helpcenterlink(self, item):
item["@type"] = "Link"
item["remoteUrl"] = item.get("url", None)
return item
def dict_hook_helpcenterreferencemanualpage(self, item):
item["@type"] = "Document"
return item
如果您在导入期间更改类型,您需要注意其他引用类型的情况。例如,集合查询(请参阅“修复无效的集合查询”)或约束(请参阅此处)。
PORTAL_TYPE_MAPPING = {
"Topic": "Collection",
"FormFolder": "EasyForm",
"HelpCenter": "Folder",
}
def global_dict_hook(self, item):
if item.get("exportimport.constrains"):
types_fixed = []
for portal_type in item["exportimport.constrains"]["locally_allowed_types"]:
if portal_type in PORTAL_TYPE_MAPPING:
types_fixed.append(PORTAL_TYPE_MAPPING[portal_type])
elif portal_type in ALLOWED_TYPES:
types_fixed.append(portal_type)
item["exportimport.constrains"]["locally_allowed_types"] = list(set(types_fixed))
types_fixed = []
for portal_type in item["exportimport.constrains"]["immediately_addable_types"]:
if portal_type in PORTAL_TYPE_MAPPING:
types_fixed.append(PORTAL_TYPE_MAPPING[portal_type])
elif portal_type in ALLOWED_TYPES:
types_fixed.append(portal_type)
item["exportimport.constrains"]["immediately_addable_types"] = list(set(types_fixed))
return item
更改工作流程
REVIEW_STATE_MAPPING = {
"internal": "published",
"internally_published": "published",
"obsolete": "private",
"hidden": "private",
}
def global_dict_hook(self, item):
if item.get("review_state") in REVIEW_STATE_MAPPING:
item["review_state"] = REVIEW_STATE_MAPPING[item["review_state"]]
return item
导出/导入注释
Plone的一些核心功能(例如评论)使用注解来存储数据。核心功能已经覆盖,但您的自定义代码或社区插件也可能使用注解。以下是迁移它们的方法。
导出:仅导出您真正需要的注解。
from zope.annotation.interfaces import IAnnotations
ANNOTATIONS_TO_EXPORT = [
"syndication_settings",
]
ANNOTATIONS_KEY = 'exportimport.annotations'
class CustomExportContent(ExportContent):
def global_dict_hook(self, item, obj):
item = self.export_annotations(item, obj)
return item
def export_annotations(self, item, obj):
results = {}
annotations = IAnnotations(obj)
for key in ANNOTATIONS_TO_EXPORT:
data = annotations.get(key)
if data:
results[key] = IJsonCompatible(data, None)
if results:
item[ANNOTATIONS_KEY] = results
return item
导入:
from zope.annotation.interfaces import IAnnotations
ANNOTATIONS_KEY = "exportimport.annotations"
class CustomImportContent(ImportContent):
def global_obj_hook(self, obj, item):
item = self.import_annotations(obj, item)
return item
def import_annotations(self, obj, item):
annotations = IAnnotations(obj)
for key in item.get(ANNOTATIONS_KEY, []):
annotations[key] = item[ANNOTATIONS_KEY][key]
return item
一些功能也在门户的注解中存储数据,例如 plone.contentrules.localassignments,plone.portlets.categoryblackliststatus,plone.portlets.contextassignments,syndication_settings。根据您的需求,您可能还想导出和导入这些内容。
导出/导入标记接口
导出:您可能只想导出所需的标记接口。在决定要迁移什么之前,检查门户中所有使用的标记接口列表是个好主意。
from zope.interface import directlyProvidedBy
MARKER_INTERFACES_TO_EXPORT = [
"collective.easyslider.interfaces.ISliderPage",
"plone.app.layout.navigation.interfaces.INavigationRoot",
]
MARKER_INTERFACES_KEY = "exportimport.marker_interfaces"
class CustomExportContent(ExportContent):
def global_dict_hook(self, item, obj):
item = self.export_marker_interfaces(item, obj)
return item
def export_marker_interfaces(self, item, obj):
interfaces = [i.__identifier__ for i in directlyProvidedBy(obj)]
interfaces = [i for i in interfaces if i in MARKER_INTERFACES_TO_EXPORT]
if interfaces:
item[MARKER_INTERFACES_KEY] = interfaces
return item
导入:
from plone.dexterity.utils import resolveDottedName
from zope.interface import alsoProvides
MARKER_INTERFACES_KEY = "exportimport.marker_interfaces"
class CustomImportContent(ImportContent):
def global_obj_hook_before_deserializing(self, obj, item):
"""Apply marker interfaces before deserializing."""
for iface_name in item.pop(MARKER_INTERFACES_KEY, []):
try:
iface = resolveDottedName(iface_name)
if not iface.providedBy(obj):
alsoProvides(obj, iface)
logger.info("Applied marker interface %s to %s", iface_name, obj.absolute_url())
except ModuleNotFoundError:
pass
return obj, item
导入时跳过版本控制
版本控制的事件处理器可能会严重影响您的导入速度。在导入之前跳过它是明智的。
VERSIONED_TYPES = [
"Document",
"News Item",
"Event",
"Link",
]
def start(self):
self.items_without_parent = []
portal_types = api.portal.get_tool("portal_types")
for portal_type in VERSIONED_TYPES:
fti = portal_types.get(portal_type)
behaviors = list(fti.behaviors)
if 'plone.versioning' in behaviors:
logger.info(f"Disable versioning for {portal_type}")
behaviors.remove('plone.versioning')
fti.behaviors = behaviors
完成所有导入和修复后,重新启用版本控制并创建初始版本,例如在 @@import_all 视图中。
from Products.CMFEditions.interfaces.IModifier import FileTooLargeToVersionError
VERSIONED_TYPES = [
"Document",
"News Item",
"Event",
"Link",
]
class ImportAll(BrowserView):
# re-enable versioning
portal_types = api.portal.get_tool("portal_types")
for portal_type in VERSIONED_TYPES:
fti = portal_types.get(portal_type)
behaviors = list(fti.behaviors)
if "plone.versioning" not in behaviors:
behaviors.append("plone.versioning")
logger.info(f"Enable versioning for {portal_type}")
if "plone.locking" not in behaviors:
behaviors.append("plone.locking")
logger.info(f"Enable locking for {portal_type}")
fti.behaviors = behaviors
transaction.get().note("Re-enabled versioning")
transaction.commit()
# create initial version for all versioned types
logger.info("Creating initial versions")
portal_repository = api.portal.get_tool("portal_repository")
brains = api.content.find(portal_type=VERSIONED_TYPES)
total = len(brains)
for index, brain in enumerate(brains):
obj = brain.getObject()
try:
portal_repository.save(obj=obj, comment="Imported Version")
except FileTooLargeToVersionError:
pass
if not index % 1000:
msg = f"Created versions for {index} of {total} items."
logger.info(msg)
transaction.get().note(msg)
transaction.commit()
msg = "Created initial versions"
transaction.get().note(msg)
transaction.commit()
处理验证错误
有时在导入过程中会得到验证错误,因为数据无法验证。这可能发生在字段中的选项由网站中的内容生成时。在这些情况下,在导入内容时,您不能确定所有选项都已在门户中存在。
这也可能发生在您有依赖于导入时不存在的内容或配置的验证器时。
有两种处理这些问题的方法
使用简单的setter绕过restapi使用的验证
将导入推迟到所有其他导入都运行完毕
使用简单的设置器
您需要指定要这样处理的内容类型和字段。
它被放在一个键中,正常导入将忽略它,并在反序列化其余数据之前使用 setattr() 设置。
SIMPLE_SETTER_FIELDS = {
"ALL": ["some_shared_field"],
"CollaborationFolder": ["allowedPartnerDocTypes"],
"DocType": ["automaticTransferTargets"],
"DPDocument": ["scenarios"],
"DPEvent" : ["Status"],
}
class CustomImportContent(ImportContent):
def global_dict_hook(self, item):
simple = {}
for fieldname in SIMPLE_SETTER_FIELDS.get("ALL", []):
if fieldname in item:
value = item.pop(fieldname)
if value:
simple[fieldname] = value
for fieldname in SIMPLE_SETTER_FIELDS.get(item["@type"], []):
if fieldname in item:
value = item.pop(fieldname)
if value:
simple[fieldname] = value
if simple:
item["exportimport.simplesetter"] = simple
def global_obj_hook_before_deserializing(self, obj, item):
"""Hook to modify the created obj before deserializing the data.
"""
# import simplesetter data before the rest
for fieldname, value in item.get("exportimport.simplesetter", {}).items():
setattr(obj, fieldname, value)
延迟导入
您也可以等待所有内容导入完毕后再设置这些字段上的值。同样,您需要找出您想这样处理哪些类型的哪些字段。
这里的数据存储在导入对象的注解中,稍后从中读取。此示例还支持使用 setattr 设置一些数据而不进行验证
from plone.restapi.interfaces import IDeserializeFromJson
from zope.annotation.interfaces import IAnnotations
from zope.component import getMultiAdapter
DEFERRED_KEY = "exportimport.deferred"
DEFERRED_FIELD_MAPPING = {
"talk": ["somefield"],
"speaker": [
"custom_field",
"another_field",
]
}
SIMPLE_SETTER_FIELDS = {"custom_type": ["another_field"]}
class CustomImportContent(ImportContent):
def global_dict_hook(self, item):
# Move deferred values to a different key to not deserialize.
# This could also be done during export.
item[DEFERRED_KEY] = {}
for fieldname in DEFERRED_FIELD_MAPPING.get(item["@type"], []):
if item.get(fieldname):
item[DEFERRED_KEY][fieldname] = item.pop(fieldname)
return item
def global_obj_hook(self, obj, item):
# Store deferred data in an annotation.
deferred = item.get(DEFERRED_KEY, {})
if deferred:
annotations = IAnnotations(obj)
annotations[DEFERRED_KEY] = {}
for key, value in deferred.items():
annotations[DEFERRED_KEY][key] = value
然后您需要在迁移中添加一个新步骤,将延迟的值从注解移动到字段
class ImportDeferred(BrowserView):
def __call__(self):
# This example reuses the form export_other.pt from collective.exportimport
self.title = "Import deferred data"
if not self.request.form.get("form.submitted", False):
return self.index()
portal = api.portal.get()
self.results = []
for brain in api.content.find(DEFERRED_FIELD_MAPPING.keys()):
obj = brain.getObject()
self.import_deferred(obj)
api.portal.show_message(f"Imported deferred data for {len(self.results)} items!", self.request)
def import_deferred(self, obj):
annotations = IAnnotations(obj, {})
deferred = annotations.get(DEFERRED_KEY, None)
if not deferred:
return
# Shortcut for simple fields (e.g. storing strings, uuids etc.)
for fieldname in SIMPLE_SETTER_FIELDS.get(obj.portal_type, []):
value = deferred.pop(fieldname, None)
if value:
setattr(obj, fieldname, value)
if not deferred:
return
# This approach validates the values and converts more complex data
deserializer = getMultiAdapter((obj, self.request), IDeserializeFromJson)
try:
obj = deserializer(validate_all=False, data=deferred)
except Exception as e:
logger.info("Error while importing deferred data for %s", obj.absolute_url(), exc_info=True)
logger.info("Data: %s", deferred)
else:
self.results.append(obj.absolute_url())
# cleanup
del annotations[DEFERRED_KEY]
显然,这个附加视图需要注册
<browser:page
name="import_deferred"
for="zope.interface.Interface"
class=".import_content.ImportDeferred"
template="export_other.pt"
permission="cmf.ManagePortal"
/>
处理 LinguaPlone 内容
导出
def global_dict_hook(self, item, obj):
# Find language of the nearest parent with a language
# Usefull for LinguaPlone sites where some content is languageindependent
parent = obj.__parent__
for ancestor in parent.aq_chain:
if IPloneSiteRoot.providedBy(ancestor):
# keep language for root content
nearest_ancestor_lang = item["language"]
break
if getattr(ancestor, "getLanguage", None) and ancestor.getLanguage():
nearest_ancestor_lang = ancestor.getLanguage()
item["parent"]["language"] = nearest_ancestor_lang
break
# This forces "wrong" languages to the nearest parents language
if "language" in item and item["language"] != nearest_ancestor_lang:
logger.info(u"Forcing %s (was %s) for %s %s ", nearest_ancestor_lang, item["language"], item["@type"], item["@id"])
item["language"] = nearest_ancestor_lang
# set missing language
if not item.get("language"):
item["language"] = nearest_ancestor_lang
# add info on translations to help find the right container
# usually this idone by export_translations
# but when migrating from LP to pam you sometimes want to check the
# tranlation info during import
if getattr(obj.aq_base, "getTranslations", None) is not None:
translations = obj.getTranslations()
if translations:
item["translation"] = {}
for lang in translations:
uuid = IUUID(translations[lang][0], None)
if uuid == item["UID"]:
continue
translation = translations[lang][0]
if not lang:
lang = "no_language"
item["translation"][lang] = translation.absolute_url()
导入
def global_dict_hook(self, item):
# Adapt this to your site
languages = ["en", "fr", "de"]
default_language = "en"
portal_id = "Plone"
# No language => lang of parent or default
if item.get("language") not in languages:
if item["parent"].get("language"):
item["language"] = item["parent"]["language"]
else:
item["language"] = default_language
lang = item["language"]
if item["parent"].get("language") != item["language"]:
logger.debug(f"Inconsistent lang: item is {lang}, parent is {item['parent'].get('language')} for {item['@id']}")
# Move item to the correct language-root-folder
# This is only relevant for items in the site-root.
# Most items containers are usually looked up by the uuid of the old parent
url = item["@id"]
parent_url = item["parent"]["@id"]
url = url.replace(f"/{portal_id}/", f"/{portal_id}/{lang}/", 1)
parent_url = parent_url.replace(f"/{portal_id}", f"/{portal_id}/{lang}", 1)
item["@id"] = url
item["parent"]["@id"] = parent_url
return item
处理无父项项的替代方法
通常,最好导出和记录无法找到容器的项目,而不是重新创建原始结构。
def update(self):
self.items_without_parent = []
def create_container(self, item):
# Override create_container to never create parents
self.items_without_parent.append(item)
def finish(self):
# export content without parents
if self.items_without_parent:
data = json.dumps(self.items_without_parent, sort_keys=True, indent=4)
number = len(self.items_without_parent)
cfg = getConfiguration()
filename = 'content_without_parent.json'
filepath = os.path.join(cfg.clienthome, filename)
with open(filepath, 'w') as f:
f.write(data)
msg = u"Saved {} items without parent to {}".format(number, filepath)
logger.info(msg)
api.portal.show_message(msg, self.request)
导出/导入 Zope 用户
默认情况下,只有存储在Plone中的用户和组会被导出/导入。您可以使用这种方法导出/导入Zope用户。
导出
from collective.exportimport.export_other import BaseExport
from plone import api
import six
class ExportZopeUsers(BaseExport):
AUTO_ROLES = ["Authenticated"]
def __call__(self, download_to_server=False):
self.title = "Export Zope users"
self.download_to_server = download_to_server
portal = api.portal.get()
app = portal.__parent__
self.acl = app.acl_users
self.pms = api.portal.get_tool("portal_membership")
data = self.all_zope_users()
self.download(data)
def all_zope_users(self):
results = []
for user in self.acl.searchUsers():
data = self._getUserData(user["userid"])
data['title'] = user['title']
results.append(data)
return results
def _getUserData(self, userId):
member = self.pms.getMemberById(userId)
roles = [
role
for role in member.getRoles()
if role not in self.AUTO_ROLES
]
# userid, password, roles
props = {
"username": userId,
"password": json_compatible(self._getUserPassword(userId)),
"roles": json_compatible(roles),
}
return props
def _getUserPassword(self, userId):
users = self.acl.users
passwords = users._user_passwords
password = passwords.get(userId, "")
return password
导入:
class ImportZopeUsers(BrowserView):
def __call__(self, jsonfile=None, return_json=False):
if jsonfile:
self.portal = api.portal.get()
status = "success"
try:
if isinstance(jsonfile, str):
return_json = True
data = json.loads(jsonfile)
elif isinstance(jsonfile, FileUpload):
data = json.loads(jsonfile.read())
else:
raise ("Data is neither text nor upload.")
except Exception as e:
status = "error"
logger.error(e)
api.portal.show_message(
u"Failure while uploading: {}".format(e),
request=self.request,
)
else:
members = self.import_members(data)
msg = u"Imported {} members".format(members)
api.portal.show_message(msg, self.request)
if return_json:
msg = {"state": status, "msg": msg}
return json.dumps(msg)
return self.index()
def import_members(self, data):
app = self.portal.__parent__
acl = app.acl_users
counter = 0
for item in data:
username = item["username"]
password = item.pop("password")
roles = item.pop("roles", [])
if not username or not password or not roles:
continue
title = item.pop("title", None)
acl.users.addUser(username, title, password)
for role in roles:
acl.roles.assignRoleToPrincipal(role, username)
counter += 1
return counter
导出/导入属性,注册设置和已安装的附加组件
当您迁移多个手动配置的类似站点时,导出和导入手动设置的配置可能很有用。
导出/导入已安装的设置和附加组件
此自定义导出导出和导入来自Plone 4.3站点的某些选择设置和插件。
导出
from collective.exportimport.export_other import BaseExport
from logging import getLogger
from plone import api
from plone.restapi.serializer.converters import json_compatible
logger = getLogger(__name__)
class ExportSettings(BaseExport):
"""Export various settings for haiku sites
"""
def __call__(self, download_to_server=False):
self.title = "Export installed add-ons various settings"
self.download_to_server = download_to_server
if not self.request.form.get("form.submitted", False):
return self.index()
data = self.export_settings()
self.download(data)
def export_settings(self):
results = {}
addons = []
qi = api.portal.get_tool("portal_quickinstaller")
for product in qi.listInstalledProducts():
if product["id"].startswith("myproject."):
addons.append(product["id"])
results["addons"] = addons
portal = api.portal.get()
registry = {}
registry["plone.email_from_name"] = portal.getProperty('email_from_name', '')
registry["plone.email_from_address"] = portal.getProperty('email_from_address', '')
registry["plone.smtp_host"] = getattr(portal.MailHost, 'smtp_host', '')
registry["plone.smtp_port"] = int(getattr(portal.MailHost, 'smtp_port', 25))
registry["plone.smtp_userid"] = portal.MailHost.get('smtp_user_id')
registry["plone.smtp_pass"] = portal.MailHost.get('smtp_pass')
registry["plone.site_title"] = portal.title
portal_properties = api.portal.get_tool("portal_properties")
iprops = portal_properties.imaging_properties
registry["plone.allowed_sizes"] = iprops.getProperty('allowed_sizes')
registry["plone.quality"] = iprops.getProperty('quality')
site_props = portal_properties.site_properties
if site_props.hasProperty("webstats_js"):
registry["plone.webstats_js"] = site_props.webstats_js
results["registry"] = json_compatible(registry)
return results
导入
导入将安装插件并在注册表中加载设置。由于Plone 5,portal_properties 已不再使用。
from logging import getLogger
from plone import api
from plone.registry.interfaces import IRegistry
from Products.CMFPlone.utils import get_installer
from Products.Five import BrowserView
from zope.component import getUtility
from ZPublisher.HTTPRequest import FileUpload
import json
logger = getLogger(__name__)
class ImportSettings(BrowserView):
"""Import various settings"""
def __call__(self, jsonfile=None, return_json=False):
if jsonfile:
self.portal = api.portal.get()
status = "success"
try:
if isinstance(jsonfile, str):
return_json = True
data = json.loads(jsonfile)
elif isinstance(jsonfile, FileUpload):
data = json.loads(jsonfile.read())
else:
raise ("Data is neither text nor upload.")
except Exception as e:
status = "error"
logger.error(e)
api.portal.show_message(
"Failure while uploading: {}".format(e),
request=self.request,
)
else:
self.import_settings(data)
msg = "Imported addons and settings"
api.portal.show_message(msg, self.request)
if return_json:
msg = {"state": status, "msg": msg}
return json.dumps(msg)
return self.index()
def import_settings(self, data):
installer = get_installer(self.context)
for addon in data["addons"]:
if not installer.is_product_installed(addon) and installer.is_product_installable(addon):
installer.install_product(addon)
logger.info(f"Installed addon {addon}")
registry = getUtility(IRegistry)
for key, value in data["registry"].items():
registry[key] = value
logger.info(f"Imported record {key}: {value}")
导出/导入注册设置
拉取请求 https://github.com/collective/collective.exportimport/pull/130 包含视图 @@export_registry 和 @@import_registry。这些视图导出和导入不使用该注册记录架构中指定默认设置的注册记录。仅导出也可能有助于确定对站点进行了哪些设置修改。
这段代码可能不会合并,但您可以在自己的项目中使用它。
将 PloneFormGen 迁移到 Easyform
要能够将 PFG 导出为 easyform,您应该在您的旧站点的 collective.easyform 中使用 migration_features_1.x 分支。Easyform 不需要安装,我们只需要 fields_model 和 actions_model 这两个方法。
导出
def dict_hook_formfolder(self, item, obj):
item["@type"] = "EasyForm"
item["is_folderish"] = False
from collective.easyform.migration.fields import fields_model
from collective.easyform.migration.actions import actions_model
# this does most of the heavy lifting...
item["fields_model"] = fields_model(obj)
item["actions_model"] = actions_model(obj)
# handle thankspage
pfg_thankspage = obj.get(obj.getThanksPage(), None)
if pfg_thankspage:
item["thankstitle"] = pfg_thankspage.title
item["thanksdescription"] = pfg_thankspage.Description()
item["showAll"] = pfg_thankspage.showAll
item["showFields"] = pfg_thankspage.showFields
item["includeEmpties"] = pfg_thankspage.includeEmpties
item["thanksPrologue"] = json_compatible(pfg_thankspage.thanksPrologue.raw)
item["thanksEpilogue"] = json_compatible(pfg_thankspage.thanksEpilogue.raw)
# optional
item["exportimport._inputStorage"] = self.export_saved_data(obj)
# Drop some PFG fields no longer needed
obsolete_fields = [
"layout",
"actionAdapter",
"checkAuthenticator",
"constrainTypesMode",
"location",
"thanksPage",
]
for key in obsolete_fields:
item.pop(key, None)
# optional: disable tabs for imported forms
item["form_tabbing"] = False
# fix some custom validators
replace_mapping = {
"request.form['": "request.form['form.widgets.",
"request.form.get('": "request.form.get('form.widgets.",
"member and member.id or ''": "member and member.getProperty('id', '') or ''",
}
# fix overrides in actions and fields to use form.widgets.xyz instead of xyz
for schema in ["actions_model", "fields_model"]:
for old, new in replace_mapping.items():
if old in item[schema]:
item[schema] = item[schema].replace(old, new)
# add your own fields if you have these issues...
for fieldname in [
"email",
"replyto",
]:
if "request/form/{}".format(fieldname) in item[schema]:
item[schema] = item[schema].replace("request/form/{}".format(fieldname), "python: request.form.get('form.widgets.{}')".format(fieldname))
return item
def export_saved_data(self, obj):
actions = {}
for data_adapter in obj.objectValues("FormSaveDataAdapter"):
data_adapter_name = data_adapter.getId()
actions[data_adapter_name] = {}
cols = data_adapter.getColumnNames()
column_count_mismatch = False
for idx, row in enumerate(data_adapter.getSavedFormInput()):
if len(row) != len(cols):
column_count_mismatch = True
logger.debug("Column count mismatch at row %s", idx)
continue
data = {}
for key, value in zip(cols, row):
data[key] = json_compatible(value)
id_ = int(time() * 1000)
while id_ in actions[data_adapter_name]: # avoid collisions during export
id_ += 1
data["id"] = id_
actions[data_adapter_name][id_] = data
if column_count_mismatch:
logger.info(
"Number of columns does not match for all rows. Some data were skipped in "
"data adapter %s/%s",
"/".join(obj.getPhysicalPath()),
data_adapter_name,
)
return actions
将导出的 PloneFormGen 数据导入到 Easyform
def obj_hook_easyform(self, obj, item):
if not item.get("exportimport._inputStorage"):
return
from collective.easyform.actions import SavedDataBTree
from persistent.mapping import PersistentMapping
if not hasattr(obj, '_inputStorage'):
obj._inputStorage = PersistentMapping()
for name, data in item["exportimport._inputStorage"].items():
obj._inputStorage[name] = SavedDataBTree()
for key, row in data.items():
obj._inputStorage[name][int(key)] = row
导出和导入 collective.cover 内容
导出
from collective.exportimport.serializer import get_dx_blob_path
from plone.app.textfield.value import RichTextValue
from plone.namedfile.file import NamedBlobImage
from plone.restapi.interfaces import IJsonCompatible
from z3c.relationfield import RelationValue
from zope.annotation.interfaces import IAnnotations
def global_dict_hook(self, item, obj):
item = self.handle_cover(item, obj)
return item
def handle_cover(self, item, obj):
if ICover.providedBy(obj):
item['tiles'] = {}
annotations = IAnnotations(obj)
for tile in obj.get_tiles():
annotation_key = 'plone.tiles.data.{}'.format(tile['id'])
annotation = annotations.get(annotation_key, None)
if annotation is None:
continue
tile_data = self.serialize_tile(annotation)
tile_data['type'] = tile['type']
item['tiles'][tile['id']] = tile_data
return item
def serialize_tile(self, annotation):
data = {}
for key, value in annotation.items():
if isinstance(value, RichTextValue):
value = value.raw
elif isinstance(value, RelationValue):
value = value.to_object.UID()
elif isinstance(value, NamedBlobImage):
blobfilepath = get_dx_blob_path(value)
if not blobfilepath:
continue
value = {
"filename": value.filename,
"content-type": value.contentType,
"size": value.getSize(),
"blob_path": blobfilepath,
}
data[key] = IJsonCompatible(value, None)
return data
导入
from collections import defaultdict
from collective.exportimport.import_content import get_absolute_blob_path
from plone.app.textfield.interfaces import IRichText
from plone.app.textfield.interfaces import IRichTextValue
from plone.namedfile.file import NamedBlobImage
from plone.namedfile.interfaces import INamedBlobImageField
from plone.tiles.interfaces import ITileType
from zope.annotation.interfaces import IAnnotations
from zope.component import getUtilitiesFor
from zope.schema import getFieldsInOrder
COVER_CONTENT = [
"collective.cover.content",
]
def global_obj_hook(self, obj, item):
if item["@type"] in COVER_CONTENT and "tiles" in item:
item = self.import_tiles(obj, item)
def import_tiles(self, obj, item):
RICHTEXT_TILES = defaultdict(list)
IMAGE_TILES = defaultdict(list)
for tile_name, tile_type in getUtilitiesFor(ITileType):
for fieldname, field in getFieldsInOrder(tile_type.schema):
if IRichText.providedBy(field):
RICHTEXT_TILES[tile_name].append(fieldname)
if INamedBlobImageField.providedBy(field):
IMAGE_TILES[tile_name].append(fieldname)
annotations = IAnnotations(obj)
prefix = "plone.tiles.data."
for uid, tile in item["tiles"].items():
# TODO: Maybe create all tiles that do not need to be defferred?
key = prefix + uid
tile_name = tile.pop("type", None)
# first set raw data
annotations[key] = item["tiles"][uid]
for fieldname in RICHTEXT_TILES.get(tile_name, []):
raw = annotations[key][fieldname]
if raw is not None and not IRichTextValue.providedBy(raw):
annotations[key][fieldname] = RichTextValue(raw, "text/html", "text/x-html-safe")
for fieldname in IMAGE_TILES.get(tile_name, []):
data = annotations[key][fieldname]
if data is not None:
blob_path = data.get("blob_path")
if not blob_path:
continue
abs_blob_path = get_absolute_blob_path(obj, blob_path)
if not abs_blob_path:
logger.info("Blob path %s for tile %s of %s %s does not exist!", blob_path, tile, obj.portal_type, obj.absolute_url())
continue
# Determine the class to use: file or image.
filename = data["filename"]
content_type = data["content-type"]
# Write the field.
with open(abs_blob_path, "rb") as myfile:
blobdata = myfile.read()
image = NamedBlobImage(
data=blobdata,
contentType=content_type,
filename=filename,
)
annotations[key][fieldname] = image
return item
修复无效的集合查询
Plone 4 和 5 之间存在一些查询变化。这修复了这些问题。
在 collective.exportimport.serializer.SerializeTopicToJson 中,将主题迁移到集合的实际迁移(目前)还没有注意这一点。
class CustomImportContent(ImportContent):
def global_dict_hook(self, item):
if item["@type"] in ["Collection", "Topic"]:
item = self.fix_query(item)
def fix_query(self, item):
item["@type"] = "Collection"
query = item.pop("query", [])
if not query:
logger.info("Drop item without query: %s", item["@id"])
return
fixed_query = []
indexes_to_fix = [
"portal_type",
"review_state",
"Creator",
"Subject",
]
operator_mapping = {
# old -> new
"plone.app.querystring.operation.selection.is":
"plone.app.querystring.operation.selection.any",
"plone.app.querystring.operation.string.is":
"plone.app.querystring.operation.selection.any",
}
for crit in query:
if crit["i"] == "portal_type" and len(crit["v"]) > 30:
# Criterion is all types
continue
if crit["o"].endswith("relativePath") and crit["v"] == "..":
# relativePath no longer accepts ..
crit["v"] = "..::1"
if crit["i"] in indexes_to_fix:
for old_operator, new_operator in operator_mapping.items():
if crit["o"] == old_operator:
crit["o"] = new_operator
if crit["i"] == "portal_type":
# Some types may have changed their names
fixed_types = []
for portal_type in crit["v"]:
fixed_type = PORTAL_TYPE_MAPPING.get(portal_type, portal_type)
fixed_types.append(fixed_type)
crit["v"] = list(set(fixed_types))
if crit["i"] == "review_state":
# Review states may have changed their names
fixed_states = []
for review_state in crit["v"]:
fixed_state = REVIEW_STATE_MAPPING.get(review_state, review_state)
fixed_states.append(fixed_state)
crit["v"] = list(set(fixed_states))
if crit["o"] == "plone.app.querystring.operation.string.currentUser":
crit["v"] = ""
fixed_query.append(crit)
item["query"] = fixed_query
if not item["query"]:
logger.info("Drop collection without query: %s", item["@id"])
return
return item
迁移到 Volto
您可以在迁移中重用 plone.volto 中的 @@migrate_to_volto 提供的迁移代码。以下示例(用于将 https://plone.org 迁移到 Volto)可以用来将任何旧版本站点迁移到 Plone 6 并使用 Volto。
您需要运行 https://github.com/plone/blocks-conversion-tool 以处理将富文本值迁移到 Volto-blocks。
有关迁移到 Volto 的更改的更多详细信息,请参阅 https://6.docs.plone.org/backend/upgrading/version-specific-migration/migrate-to-volto.html。
from App.config import getConfiguration
from bs4 import BeautifulSoup
from collective.exportimport.fix_html import fix_html_in_content_fields
from collective.exportimport.fix_html import fix_html_in_portlets
from contentimport.interfaces import IContentimportLayer
from logging import getLogger
from pathlib import Path
from plone import api
from plone.volto.browser.migrate_to_volto import migrate_richtext_to_blocks
from plone.volto.setuphandlers import add_behavior
from plone.volto.setuphandlers import remove_behavior
from Products.CMFPlone.utils import get_installer
from Products.Five import BrowserView
from zope.interface import alsoProvides
import requests
import transaction
logger = getLogger(__name__)
DEFAULT_ADDONS = []
class ImportAll(BrowserView):
def __call__(self):
request = self.request
# Check if Blocks-conversion-tool is running
headers = {
"Accept": "application/json",
"Content-Type": "application/json",
}
r = requests.post(
"https://127.0.0.1:5000/html", headers=headers, json={"html": "<p>text</p>"}
)
r.raise_for_status()
# Submit a simple form template to trigger the import
if not request.form.get("form.submitted", False):
return self.index()
portal = api.portal.get()
alsoProvides(request, IContentimportLayer)
installer = get_installer(portal)
if not installer.is_product_installed("contentimport"):
installer.install_product("contentimport")
# install required add-ons
for addon in DEFAULT_ADDONS:
if not installer.is_product_installed(addon):
installer.install_product(addon)
# Fake the target being a classic site even though plone.volto is installed...
# 1. Allow Folders and Collections (they are disabled in Volto by default)
portal_types = api.portal.get_tool("portal_types")
portal_types["Collection"].global_allow = True
portal_types["Folder"].global_allow = True
# 2. Enable richtext behavior (otherwise no text will be imported)
for type_ in ["Document", "News Item", "Event"]:
add_behavior(type_, "plone.richtext")
transaction.commit()
cfg = getConfiguration()
directory = Path(cfg.clienthome) / "import"
# Import content
view = api.content.get_view("import_content", portal, request)
request.form["form.submitted"] = True
request.form["commit"] = 500
view(server_file="Plone.json", return_json=True)
transaction.commit()
# Run all other imports
other_imports = [
"relations",
"members",
"translations",
"localroles",
"ordering",
"defaultpages",
"discussion",
"portlets", # not really useful in Volto
"redirects",
]
for name in other_imports:
view = api.content.get_view(f"import_{name}", portal, request)
path = Path(directory) / f"export_{name}.json"
if path.exists():
results = view(jsonfile=path.read_text(), return_json=True)
logger.info(results)
transaction.get().note(f"Finished import_{name}")
transaction.commit()
else:
logger.info(f"Missing file: {path}")
# Optional: Run html-fixers on richtext
fixers = [anchor_fixer]
results = fix_html_in_content_fields(fixers=fixers)
msg = "Fixed html for {} content items".format(results)
logger.info(msg)
transaction.get().note(msg)
transaction.commit()
results = fix_html_in_portlets()
msg = "Fixed html for {} portlets".format(results)
logger.info(msg)
transaction.get().note(msg)
transaction.commit()
view = api.content.get_view("updateLinkIntegrityInformation", portal, request)
results = view.update()
msg = f"Updated linkintegrity for {results} items"
logger.info(msg)
transaction.get().note(msg)
transaction.commit()
# Rebuilding the catalog is necessary to prevent issues later on
catalog = api.portal.get_tool("portal_catalog")
logger.info("Rebuilding catalog...")
catalog.clearFindAndRebuild()
msg = "Finished rebuilding catalog!"
logger.info(msg)
transaction.get().note(msg)
transaction.commit()
# This uses the blocks-conversion-tool to migrate to blocks
logger.info("Start migrating richtext to blocks...")
migrate_richtext_to_blocks()
msg = "Finished migrating richtext to blocks"
transaction.get().note(msg)
transaction.commit()
# Reuse the migration-form from plon.volto to do some more tasks
view = api.content.get_view("migrate_to_volto", portal, request)
# Yes, wen want to migrate default pages
view.migrate_default_pages = True
view.slate = True
logger.info("Start migrating Folders to Documents...")
view.do_migrate_folders()
msg = "Finished migrating Folders to Documents!"
transaction.get().note(msg)
transaction.commit()
logger.info("Start migrating Collections to Documents...")
view.migrate_collections()
msg = "Finished migrating Collections to Documents!"
transaction.get().note(msg)
transaction.commit()
reset_dates = api.content.get_view("reset_dates", portal, request)
reset_dates()
transaction.commit()
# Disallow folders and collections again
portal_types["Collection"].global_allow = False
portal_types["Folder"].global_allow = False
# Disable richtext behavior again
for type_ in ["Document", "News Item", "Event"]:
remove_behavior(type_, "plone.richtext")
return request.response.redirect(portal.absolute_url())
def anchor_fixer(text, obj=None):
"""Remove anchors since they are not supported by Volto yet"""
soup = BeautifulSoup(text, "html.parser")
for link in soup.find_all("a"):
if not link.get("href") and not link.text:
# drop empty links (e.g. anchors)
link.decompose()
elif not link.get("href") and link.text:
# drop links without a href but keep the text
link.unwrap()
return soup.decode()
使用 collective.jsonify 创建的数据迁移到非常旧的 Plone 版本
版本低于 Plone 4 的版本不支持 plone.restapi,而这是 collective.exportimport 所必需的。
要将 Plone 1、2 和 3 迁移到 Plone 6,您可以使用 collective.jsonify 进行导出,并使用 collective.exportimport 进行导入。
使用 collective.jsonify 导出
使用 https://github.com/collective/collective.jsonify 导出内容。
您可以通过使用 外部方法 包括 collective.jsonify 的方法。有关更多信息,请参阅 https://github.com/collective/collective.jsonify/blob/master/docs/install.rst。
为了更好地与 collective.exportimport 一起工作,您可以使用 additional_wrappers 功能扩展导出的数据。添加有关项目父项的信息,以便 collective.exportimport 更容易导入数据。
以下是 json_methods.py 的完整示例,它应该位于 BUILDOUT_ROOT/parts/instance/Extensions/
def extend_item(obj, item):
"""Extend to work better well with collective.exportimport"""
from Acquisition import aq_parent
parent = aq_parent(obj)
item["parent"] = {
"@id": parent.absolute_url(),
"@type": getattr(parent, "portal_type", None),
}
if getattr(parent.aq_base, "UID", None) is not None:
item["parent"]["UID"] = parent.UID()
return item
以下是 json_methods.py 的完整示例,它应该位于 <BUILDOUT_ROOT>/parts/instance/Extensions/
from collective.jsonify.export import export_content as export_content_orig
from collective.jsonify.export import get_item
EXPORTED_TYPES = [
"Folder",
"Document",
"News Item",
"Event",
"Link",
"Topic",
"File",
"Image",
"RichTopic",
]
EXTRA_SKIP_PATHS = [
"/Plone/archiv/",
"/Plone/do-not-import/",
]
# Path from which to continue the export.
# The export walks the whole site respecting the order.
# It will ignore everything untill this path is reached.
PREVIOUS = ""
def export_content(self):
return export_content_orig(
self,
basedir="/var/lib/zope/json",
skip_callback=skip_item,
extra_skip_classname=[],
extra_skip_id=[],
extra_skip_paths=EXTRA_SKIP_PATHS,
batch_start=0,
batch_size=10000,
batch_previous_path=PREVIOUS or None,
)
def skip_item(item):
"""Return True if the item should be skipped"""
portal_type = getattr(item, "portal_type", None)
if portal_type not in EXPORTED_TYPES:
return True
def extend_item(obj, item):
"""Extend to work better well with collective.exportimport"""
from Acquisition import aq_parent
parent = aq_parent(obj)
item["parent"] = {
"@id": parent.absolute_url(),
"@type": getattr(parent, "portal_type", None),
}
if getattr(parent.aq_base, "UID", None) is not None:
item["parent"]["UID"] = parent.UID()
return item
要使用这些示例,在 ZMI 根目录的 Zope 根处创建三个“外部方法”
id: “export_content”,module name: “json_methods”,function name: “export_content”
id: “get_item”,module name: “json_methods”,function name: “get_item”
id: “extend_item”,module name: “json_methods”,function name: “extend_item”
然后您可以通过查询字符串将扩展器传递给导出:https://127.0.0.1:8080/Plone/export_content?additional_wrappers=extend_item
使用 collective.jsonify 导入
有两个问题需要解决,以允许 collective.exportimport 导入由 collective.jsonify 生成的数据。
数据位于目录中,而不是在一个大型的 json 文件中。
json 的格式不是预期的。
从版本 1.8 开始,您可以传递一个迭代器到导入中。
您需要创建一个目录遍历器,按正确的方式排序 json 文件。默认情况下,它将按顺序导入它们:1.json、10.json、100.json、101.json 等。
from pathlib import Path
def filesystem_walker(path=None):
root = Path(path)
assert(root.is_dir())
folders = sorted([i for i in root.iterdir() if i.is_dir() and i.name.isdecimal()], key=lambda i: int(i.name))
for folder in folders:
json_files = sorted([i for i in folder.glob("*.json") if i.stem.isdecimal()], key=lambda i: int(i.stem))
for json_file in json_files:
logger.debug("Importing %s", json_file)
item = json.loads(json_file.read_text())
item["json_file"] = str(json_file)
item = prepare_data(item)
if item:
yield item
行走者将路径设置为根目录,其中包含一个或多个存放json文件的目录。文件排序是使用文件名中的数字完成的。
prepare_data 方法在将数据传递给导入之前修改数据。在导出期间,collective.exportimport 执行一个非常类似的任务。
def prepare_data(item):
"""modify jsonify data to work with c.exportimport"""
# Drop relationfields or defer the import
item.pop("relatedItems", None)
mapping = {
# jsonify => exportimport
"_uid": "UID",
"_type": "@type",
"_path": "@id",
"_layout": "layout",
# AT fieldnames => DX fieldnames
"excludeFromNav": "exclude_from_nav",
"allowDiscussion": "allow_discussion",
"subject": "subjects",
"expirationDate": "expires",
"effectiveDate": "effective",
"creation_date": "created",
"modification_date": "modified",
"startDate": "start",
"endDate": "end",
"openEnd": "open_end",
"eventUrl": "event_url",
"wholeDay": "whole_day",
"contactEmail": "contact_email",
"contactName": "contact_name",
"contactPhone": "contact_phone",
"imageCaption": "image_caption",
}
for old, new in mapping.items():
item = migrate_field(item, old, new)
if item.get("constrainTypesMode", None) == 1:
item = migrate_field(item, "constrainTypesMode", "constrain_types_mode")
else:
item.pop("locallyAllowedTypes", None)
item.pop("immediatelyAddableTypes", None)
item.pop("constrainTypesMode", None)
if "id" not in item:
item["id"] = item["_id"]
return item
def migrate_field(item, old, new):
if item.get(old, _marker) is not _marker:
item[new] = item.pop(old)
return item
您可以将生成器 filesystem_walker 传递给导入。
class ImportAll(BrowserView):
def __call__(self):
# ...
cfg = getConfiguration()
directory = Path(cfg.clienthome) / "import"
# import content
view = api.content.get_view("import_content", portal, request)
request.form["form.submitted"] = True
request.form["commit"] = 1000
view(iterator=filesystem_walker(directory / "mydata"))
# import default-pages
import_deferred = api.content.get_view("import_deferred", portal, request)
import_deferred()
class ImportDeferred(BrowserView):
def __call__(self):
self.title = "Import Deferred Settings (default pages)"
if not self.request.form.get("form.submitted", False):
return self.index()
for brain in api.content.find(portal_type="Folder"):
obj = brain.getObject()
annotations = IAnnotations(obj)
if DEFERRED_KEY not in annotations:
continue
default = annotations[DEFERRED_KEY].pop("_defaultpage", None)
if default and default in obj:
logger.info("Setting %s as default page for %s", default, obj.absolute_url())
obj.setDefaultPage(default)
if not annotations[DEFERRED_KEY]:
annotations.pop(DEFERRED_KEY)
api.portal.show_message("Done", self.request)
return self.index()
collective.jsonify 将关于关系、翻译和默认页面的信息放入导出文件中。您可以使用这种方法将导入推迟,以在所有项目导入后处理该数据。上面的示例 ImportDeferred 使用这种方法设置默认页面。
下面的 global_obj_hook 将数据存储在注释中。
def global_obj_hook(self, obj, item):
# Store deferred data in an annotation.
keys = ["_defaultpage"]
data = {}
for key in keys:
if value := item.get(key, None):
data[key] = value
if data:
annotations = IAnnotations(obj)
annotations[DEFERRED_KEY] = data
翻译
本产品已翻译成
西班牙语
贡献
支持
如果您遇到问题,请告诉我们。
许可证
本项目遵循GPLv2许可。
作者
贡献者
Philip Bauer,bauer@starzel.de
Maurits van Rees,m.van.rees@zestsoftware.nl
Fred van Dijk,f.van.dijk@zestsoftware.nl
Leonardo J. Caballero G.,leonardocaballero@gmail.com
变更日志
1.12 (2024-03-08)
修复迁移块以使Volto站点可移植并支持plone.distribution。[pbauer, tlotze]
1.11 (2024-02-28)
修复使用设置“将blob作为blob路径包含”时导出具有plone.namedfile.file.NamedFile属性的对象(因此不是blob)的AtributeError: 'NamedFile' object has no attribute '_blob'。[valipod]
添加更多的Python 2兼容版本规范并更新README。[thet]
修复导入内容时工作流没有time变量时的KeyError: time。[maurits]
允许在不应用默认html_fixer的情况下使用fix_html_in_content_fields。[pbauer]
尝试在导出内容时恢复损坏的blob。[thet]
在导出到单独的JSON文件时,也将错误写入单独的errors.json文件。这修复了导出末尾的错误和没有错误写入的问题。[thet]
添加对ATTopic export_content的支持。[avoinea]
在导入期间将主体添加到已存在的组中(#228)[pbauer]
在导出成员时忽略组的传递成员(#240)[pbauer]
1.10 (2023-10-11)
迁移端口令数据时不要重复使用mapping变量。[witsch]
修复编辑修订作者 - 参考编号 #216 [avoinea]
更好地支持门户导入,避免两次解析JSON。[gotcha]
在网站根目录迁移端口令。[ThibautBorn]
支持导出和导入,使每个内容项都有一个单独的json文件。[pbauer]
1.9 (2023-05-18)
允许为导出传递自定义文件名。[pbauer]
支持导出和导入Plone站点根(使用更新策略)。[pbauer]
修复当连接使用TmpStore时blob导出。[gotcha, pbauer]
修复端口令richtext字段导入。[mpeeters]
添加端口令在导出数据上的位置。[mpeeters]
将使用plone4中路径的端口令的根迁移到使用uid(导航、搜索、事件、集合)。[pbauer]
使讨论和端口令的导出具有上下文。[mpeeters]
修复导入组时的关键错误:不要将一个组属于的组作为新组成员导入。这可能导致组拥有比应有的更多权限。[pbauer]
1.8 (2023-04-20)
导入:在我们调用自定义钩子之前运行set_uuid方法,以便钩子可以访问项目UUID。修复 #185。[pbauer]
在README中记录COLLECTIVE_EXPORTIMPORT_CENTRAL_DIRECTORY。[fredvd]
添加西班牙语翻译。[macagua]
添加国际化支持。[macagua]
修复html:改进从缩放到图片变体的映射。[maurits]
允许在img_variant_fixer中覆盖回退变体。默认使用‘medium’。[maurits]
让fix_html视图能够在当前上下文中工作。[maurits]
修复获取blob路径的方式。(#180)[ale-rt]
当文档是文件夹时,创建容器文档以包含没有父级的项。[JeffersonBledsoe]
添加支持将任何迭代器作为导入的数据源。[pbauer]
为文档添加导入collective.jsonify数据的示例。[pbauer]
改进主题的序列化:- 使用Plone 5中添加的新标准- 为某些标准添加回退- 导出sort_on和sort_reversed- 导出customView为tabular_view [pbauer]
无论特定内容对象是否启用了讨论支持,始终导入讨论。[ajung]
1.7 (2023-01-20)
在内容类型导出列表中过滤掉“讨论项”。评论有自己的导出和导入视图。对评论进行正常内容类型导出会在尝试找到父级时引发KeyError。(#112)[fredvd]
在导入_translation端点条件中更加具体,以安装p.a.multilingual 1.x的站点。[erral]
修复导入隐藏部件为可见的问题。(#152)[pbauer]
在查询TranslationGroup项时使用Language=all。[erral]
通过处理已存在的成员来修复成员导入。[sunew]
不使用new_id,因为钩子可以更改item["id"]。[pbauer]
支持导出blob路径,即使没有访问blob的权限。[pbauer]
当在Plone 6中运行@@fix_html时,在html字段中设置图像变体。[pbauer]
1.6 (2022-10-07)
导出和导入所有组成员(包括ldap用户和组)。之前它只导出在Plone中创建的用户和组。[pbauer]
支持导入没有UUID的内容(例如,从外部源导入)。所需的最小数据是@id,@type,id和@parent[“@id”]。\[pbauer\]
在序列化基于词汇的字段时,仅导出值而不是token/title。[pbauer]
改进导入过程中的错误日志。[pbauer]
添加INCLUDE_PATHS以指定仅应导入哪些路径。[pbauer]
添加import_review_state以允许在导入期间覆盖设置review_state。[pbauer]
导出父级UID并使用它来找到导入容器。[pbauer]
将各种导出钩子移动到update_export_data以提高可读性。[pbauer]
通过传递download_to_server=True支持将所有导出传递到服务器(#115)。[pbauer]
添加对将自定义html-fixers添加到fix_html_in_content_fields的支持。[pbauer]
1.5 (2022-04-26)
修复检查父级时getPhysicalPath的AttributeError,问题123。[maurits]
导出和导入重定向工具数据。[gotcha, Michael Penninck]
将Products.TALESField字段序列化为原始值而不是评估表达式。(适用于导出PFG覆盖)[sauzher]
确保我们永远不会更改获取的修改日期或创建日期。[pbauer]
导出和导入工作流历史。[pbauer]
在导入部件时发生错误时优雅地失败。[pbauer]
忽略应导入内容的非文件夹容器。[pbauer]
使用目录代替ZopeFindAndApply,并改进export_discussion的日志。[pbauer]
添加长整数的转换器(仅py2)。[pbauer]
默认不导出链接完整性关系。[pbauer]
当导出内容失败时记录详细的异常。[pbauer]
为内容导出添加开始和完成钩子。[pbauer]
重写默认页面的导出/导入:使用默认页面的uuid而不是id。重写获取default_page以修复翻译内容的各种问题。[pbauer]
添加内容的版本/修订版本的导出和导入(#105)。[pbauer]
1.4 (2022-01-07)
修复debug标志在ExportRelations中的问题。[petschki]
使用restapi反序列化部件数据以修复导入RichText。[pbauer]
修复导入带html实体的richtext。修复#99 [pbauer]
通过使用自定义find_object来保留对浏览器视图的链接。修复#97 [pbauer]
在导入具有替换策略的项目时忽略链接完整性。[pbauer]
为fix_html添加测试。[pbauer]
1.3 (2021-12-08)
处理网站根对象的默认页面。[fulv]
可选(复选框)在导入时跳过现有内容,而不是用随机ID生成新内容。[petschki]
当调用import_content并使用return_json和server_file时修复UnboundLocalError。[petschki]
添加每导入x个项目提交一次的选项。[pbauer]
在各种情况下改进导入过程中的日志记录。[pbauer]
处理api.content.get(path=parent_path)引发NotFound而不是返回None的情况。[pbauer]
保持import_to_current_folder的值。[pbauer]
修复py3中的html未转义问题。[pbauer]
修复ATNewsItem图像字段内容的序列化。[gotcha]
将eventUrl迁移到event_url(从AT到DX)。[ThibautBorn]
在导出时记录无法序列化的项,而不是中止导出。[ThibautBorn]
为export_localroles添加item_hook。[ThibautBorn]
修复skip_existing_content和import_to_current_folder的复选框处理。[pbauer]
将中间提交代码移动到commit_hook方法中以便覆盖。[pbauer]
添加hook global_obj_hook_before_deserializing,在反序列化数据之前修改创建的对象。[pbauer]
添加在导入时更新和替换现有内容的功能(#76)[pbauer]
导入本地角色后重新索引权限。[pbauer]
添加导出/导入约束的功能,但导入内容时不检查约束或权限(#71)。[pbauer]
1.2 (2021-10-11)
防止在同一个数据库中创建不同Plone站点的内容。[maurits]通常,在本地主机上开发时清理父路径。
读取环境变量COLLECTIVE_EXPORTIMPORT_CENTRAL_DIRECTORY(#51)。当设置时,用于存储导出文件和获取导入文件。这在同一服务器上的多个Plone站点之间共享内容时很有用。[maurits]
在导入评论时取消转义HTML实体和换行符。[pbauer]
导出和导入具有可配置类型、深度和路径的完整站点或内容树(#40)。[pbauer]
添加将blob作为blob路径导出的选项。[pbauer, maurits]
修复创建缺失文件夹结构的问题。[maurits]
导出和导入组件(#39)。[pbauer]
使用生成器/yield将内容导出到文件。这避免了导出文件大小的内存膨胀(#41)。[fredvd]
1.1 (2021-08-02)
添加从服务器导入文件的选项。[maurits]
在export_content.pt中缺少</form>关闭标签。[petschki]
支持在导出/导入本地角色时禁用本地角色的禁用获取。[pbauer]
使用unrestrictedSearchResults实际导出所有内容。[pbauer]
在导入一种类型后添加提交信息。[pbauer]
修复一些情况下获取容器的问题。[pbauer]
修复在不使用dexterity、zc.relation或plone.app.contenttypes的情况下在Plone 4.3中使用的问题。[pbauer]
修复集合和子集合父代的@id。修复#30 [pbauer]
修复在使用dexterity但未使用z3c.relationfield的情况下在Plone 4.3中使用的问题。[maurits]
添加导出和导入讨论/评论。[pbauer]
添加导入后修复集合查询的选项。[thomasmassmann]
重置创建日期。修复#29 [pbauer]
由于与restapi的ConfigurationConflictError,删除自定义关系序列化器。无论如何,在使用默认设置时,在update_data_for_migration中关系都会被丢弃。[pbauer]
迁移主题的批量大小。[pbauer]
修复找不到项容器时重用以前容器的問題。[pbauer]
添加在导入一个文件后执行的hook self.finish()。[pbauer]
修复使用较旧版本的setuptools的安装问题(#35)[pbauer]
修复使用pip的安装问题(#36)[ericof]
不要将可导出FTIs限制为允许导出类型为CalendarXFolder或ATTopic Criteria。[pbauer]
添加在导入一个文件后执行的hook self.start()。[pbauer]
1.0 (2021-04-27)
在导入时创建实例时,支持使用 factory_kwargs 设置值。这可以用于设置在订阅 IObjectAddedEvent 时需要存在的值。[pbauer]
1.0b1 (2021-03-26)
添加将导出保存到服务器的选项。[pbauer]
修复 import_relations 和 import_ordering 中的问题。[pbauer]
在 export_content 中使用链接到其他导出,以便更容易覆盖。[pbauer]
添加导出 LinguaPlone 翻译的支持。[pbauer]
1.0a2 (2021-03-11)
简化包结构并删除所有不必要的文件 [pbauer]
添加父级中位置的导出/导入功能 [pbauer]
1.0a1 (2021-03-10)
初始发布。 [pbauer]
项目详情
下载文件
下载适合您平台的文件。如果您不确定选择哪个,请了解更多关于 安装包 的信息。
源分发
构建分发
集体出口/进口的散列 - 1.12.tar.gz
算法 | 散列摘要 | |
---|---|---|
SHA256 | 4f35d8426df696b13d23c87cc2a4767cd617f52fb8d0f1d00b5229a5cf329a3d |
|
MD5 | 81b87e5d3c2e544652d46df9f7d5677e |
|
BLAKE2b-256 | a1aa40140443bf647e9ea87ed442cf15c6c10533bd96d92786fd2b9422897288 |
集体出口/进口的散列 - 1.12-py3-none-any.whl
算法 | 散列摘要 | |
---|---|---|
SHA256 | 3ada9bf517523715568ddedc825f08f4c7f50ed805b7e689e5f2536535764c90 |
|
MD5 | 3d12826021e607c208da0ddb7fe6fad3 |
|
BLAKE2b-256 | a30663f8666c666105e353c11e9f21985c8d6f0d4b5158bba1f0b2ecad9ae003 |